[fix](fe) Check all aggregate function arguments#65537
Open
morrySnow wants to merge 1 commit into
Open
Conversation
morrySnow
requested review from
924060929,
englefly and
starocean999
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I found one correctness issue in the aggregate-parameter validation.
Critical checkpoints:
- Goal: the PR intends to reject aggregate functions nested in aggregate parameters. It covers the simple non-distinct
group_concat(k order by sum(k))case, but the validation still runs after some aggregate-containing children can be normalized to slots. - Scope/focus: the code change is small, but the affected normalization path has parallel distinct and pushdown-triggered order-key branches.
- Concurrency/lifecycle/config/persistence/FE-BE protocol: not involved.
- Parallel paths: issue found in distinct non-slot children and non-distinct order keys containing pushdown-triggering expressions.
- Tests: the new FE and regression tests use the right expected-error pattern, but they only cover the simple non-distinct order-key case and miss the variants in the inline comment.
- User focus: no additional user-provided focus was supplied.
Validation was static only. I did not run builds or tests because this review prompt forbids builds/code modifications, and this checkout is not worktree-initialized and lacks thirdparty/installed / thirdparty/installed/bin/protoc.
| bottomSlotContext.normalizeToUseSlotRef(SessionVarGuardExpr.getExprWithGuard(aggFuncs)); | ||
| if (normalizedAggFuncs.stream().anyMatch(agg -> !agg.children().isEmpty() | ||
| && agg.child(0).containsType(AggregateFunction.class))) { | ||
| && agg.children().stream().anyMatch(e -> e.containsType(AggregateFunction.class)))) { |
Contributor
There was a problem hiding this comment.
This check still runs too late for children that were put into needPushDownSelfExprs. For example, group_concat(distinct k order by sum(k)) unwraps the order key in the distinct branch and maps sum(k) to a bottom-project alias slot before this block runs; similarly, count(distinct sum(k)) or an order key like sum(k) + row_number() over(...) can be replaced by a slot first. After normalizeToUseSlotRef, the aggregate child no longer contains an AggregateFunction, so this check misses the nested aggregate and may only fail later as an aggregate pushed into the wrong plan node. Please validate the original aggregate children/order-key children for non-window aggregate functions before any slot normalization, and add coverage for the distinct and pushdown-triggered variants.
Contributor
TPC-H: Total hot run time: 29293 ms |
Contributor
TPC-DS: Total hot run time: 180871 ms |
Contributor
ClickBench: Total hot run time: 25.04 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
morrySnow
force-pushed
the
fix-normalize-agg
branch
from
f5b93be to
9804ea6
Compare
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Requesting changes for one correctness regression in NormalizeAggregate.
Critical checkpoint conclusions:
- Goal: the PR aims to reject nested aggregate parameters earlier, but the current implementation also rejects valid top-level aggregates.
- Scope: the patch is small and localized, but the validation scope is too broad.
- Nereids rule behavior:
CollectNonWindowedAggFuncsWithSessionVarcollects aggregate roots, and the helper used by the new guard treats those roots as aggregate matches. - Tests: the added FE and regression tests cover the intended
group_concat(k order by sum(k))error shape, but I did not run tests because this review runner is static-only and lacksthirdparty/installed/protoc. - Existing review context: I did not duplicate the existing thread about validating original distinct/order-key children before slot normalization.
- Concurrency, lifecycle, config, persistence, FE-BE protocol, cloud/shared-nothing, and observability: not involved in this patch.
- User focus: no additional user-provided focus was present.
Round 2 convergence returned NO_NEW_VALUABLE_FINDINGS; the only accepted inline issue is below.
Issue Number: None
Related PR: None
Problem Summary: NormalizeAggregate only checked the first child of each aggregate function for nested aggregate functions. Aggregate functions in later arguments, such as SUM in the ORDER BY argument of GROUP_CONCAT, could bypass analysis validation. Check every aggregate child and add FE unit and regression coverage for the invalid query.
Reject nested aggregate functions in every aggregate function argument.
- Test: Unit Test and Regression test
- Unit Test: NormalizeAggregateTest
- Regression test: query_p0/aggregate/agg_group_concat
- Behavior changed: Yes, invalid nested aggregate functions in non-first arguments are rejected during analysis
- Does this need documentation: No
morrySnow
force-pushed
the
fix-normalize-agg
branch
from
9804ea6 to
a4ad850
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Reviewed the current PR head a4ad85013cd35b76ced1b7054c6a7475986c396d against the authoritative diff and existing review threads. I did not find a new non-duplicate issue to raise.
Checkpoint conclusions:
- Goal: the patch moves nested aggregate validation before NormalizeAggregate slot normalization and now checks aggregate children/order keys before alias replacement.
- Scope: the change is small and focused in NormalizeAggregate plus one FE unit test and one regression exception test.
- Correctness: ordinary aggregates are not rejected because the aggregate root is no longer checked; nested non-window aggregates in arguments and group_concat ORDER BY keys are detected through the child scan; window aggregate order-key cases remain valid.
- Parallel paths: distinct and group_concat ORDER BY paths use the same children/OrderExpression representation, and remaining explicit distinct/pushdown coverage concerns are already covered by existing thread
discussion_r3569517204. - Tests: added FE unit and regression negative coverage for
group_concat(k order by sum(k)); no result.outupdate is needed for the regressiontest { exception ... }block. - Concurrency/lifecycle/config/persistence/FE-BE protocol/data writes/observability: not materially involved by this FE analysis-rule change.
- Performance: added traversal is limited to collected aggregate child expressions and is not a new hot-path concern.
No additional user-provided review focus was supplied.
Validation note: static review only. I did not run builds or tests because this review prompt forbids build attempts, and this checkout also lacks thirdparty/installed/bin/protoc.
Contributor
TPC-H: Total hot run time: 29388 ms |
Contributor
TPC-DS: Total hot run time: 177594 ms |
Contributor
ClickBench: Total hot run time: 24.97 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: NormalizeAggregate only checked the first child of each aggregate function for nested aggregate functions. Aggregate functions in later arguments, such as SUM in the ORDER BY argument of GROUP_CONCAT, could bypass analysis validation. Check every aggregate child and add FE unit and regression coverage for the invalid query.
Release note
Reject nested aggregate functions in every aggregate function argument.
Check List (For Author)